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The statistical properties of nonlinear phase noise, often called the Gordon-Mollenauer effect, is 
studied analytically when the number of fiber spans is very large. The joint characteristic functions 
of the nonlinear phase noise with electric field, received intensity, and the phase of amplifier noise are 
all derived analytically. Based on the joint characteristic function of nonlinear phase noise with the 
phase of amplifier noise, the error probability of signal having nonlinear phase noise is calculated 
using the Fourier series expansion of the probability density function. The error probability is 
increased due to the dependence between nonlinear phase noise and the phase of amplifier noise. 
When the received intensity is used to compensate the nonlinear phase noise, the optimal linear 
and nonlinear minimum mean-square error compensators are derived analytically using the joint 
characteristic function of nonlinear phase noise and received intensity. Using the joint probability 
density of received amplitude and phase, the optimal maximum a posteriori probability detector is 
derived analytical. The nonlinear compensator always performs better than linear compensator. 
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I. INTRODUCTION 

When optical amplifiers are used to compensate for 
fiber loss, the interaction of amplifier noises and the fiber 
Kerr effect causes nonlinear phase noise, often called 
the Gordon-Mollenauer effect or more precisely, self- 
phase modulation induced nonlinear phase noise. Non- 
linear phase noise degrades phase-modulated signal like 
phase-shifted keying (PSK) and differential phase-shift 
keying (DPSK) signal H S 1 S I 0, SH M M 
[jjj . This class of constant-intensity modulation has 
renewed attention recently for l ong haul a nd / or spectral 
efficiency transmission systems [lj, [lj, [la, UJl UJI UM, l2ii 
Him EHM Ellll, mostly DPSK signal using return- 
to-zero (RZ) pulses or differential quad rature phase-shift 
keying (DQPSK) signal [H |U El ll |U E3 • The com- 
parison of DPSK to on-off keying signal shows advantage 
of DPSK signal in certain applications fA HTl l3ll l3^| . 

Traditionally, the performance of a system with nonlin- 
ear phase noise is evaluated based on the phase variance 
QlaSmmiMIIlor spectral broadening USSHH. 
However, it is found that the nonlinear phase noise is not 
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Gaussian-distributed both experimentally 6j and ana- 
lytically 0,m,|33. For non-Gaussian noise, neither the 
variance nor the Q-factor [H is sufficient to charac- 
terize the performance of the system. The probability 
density function (p.d.f.) is necessary to better under- 
stand the noise properties and evaluates the system per- 
formance. 

The p.d.f. of nonlinear phase noise alone IS 
not sufficient to characterize the signal with nonlinear 
phase noise. Because of the dependence between nonlin- 
ear phase noise and signal phase, the joint p.d.f. of the 
nonlinear phase noise and the signal phase is necessary 
to find the error probability for a phase-modulated sig- 
nal. This article provides analytical expressions of the 
joint asymptotic characteristic functions of the nonlinear 
phase noise and the received electric field without non- 
linear phase noise. The amplifier noise is asymptotically 
modeled as a distributed process for a large number of 
fiber spans. After the characteristic function is derived 
analytically, the p.d.f. is the inverse Fourier transform 
of the corresponding characteristic function. The depen- 
dence between nonlinear phase noise and the phase of 
amplifier noise increases the error probability. 

The received phase is the summation of the nonlinear 
phase noise and the phase of amplifier noise. Although 
it is obvious that nonlinear phase noise is uncorrelated 
with the phase of amplifier noise 0, |t| , as non-Gaussian 
random variables, they are weakly depending on each 
other. Using the joint characteristic function of nonlinear 
phase noise and the phase of amplifier noise, the p.d.f. of 
the received phase can be expanded as a Fourier series. 
Using the Fourier series, the error probability of PSK and 
DPSK signal is evaluated by a series summation. Because 
the nonlinear phase noise has a weak dependence on the 
phase of amplifier noise, the Fourier series expansion is 
more complicated than traditional method in which the 
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extra phase noise is independent of the signal phase 0, 
Elj or the approximation of [1,0 . For PSK signals, in 
contrary to 01, the received phase does not distribute 
symmetrically with respect to the mean nonlinear phase 
shift. 

Correlated with each other, the received intensity can 
be used to compensate the nonlinear phase noise. When 
a linear compensator compensates the nonlinear phase 
noise u sing a scaled version of the received intensity 
[33l l34l l35l l42l| , the optimal linear compensator to min- 
imize the variance of the residual nonlinear phase noise 
is found using the joint characteristic function of non- 
linear phase noise and received intensity. However, as 
the nonlinear phase noise is not Gaussian distributed, 
the minimum mean-square linear compensator does not 
necessary minimize the error probability of the compen- 
sated signals. When the exact error probability of lin- 
early compensated signals is derived, the optimal linear 
compensator can be found using numerical optimization. 

The minimum mean-square error (MMSE) compen- 
sator is the conditional mean of the nonlinear phase noise 
given the received intensity [SI Sec. 10.2]. Using the 
conditional characteristic function of the nonlinear phase 
noise given the received intensity, the optimal nonlinear 
compensator is found to perform slightly better than the 
linear compensator. The joint p.d.f. of the received am- 
plitude and phase can also express as a Fourier series 
with Fourier coefficients depending on the received am- 
plitude. Using the joint p.d.f. of the received amplitude 
and phase, the optimal detector and the corresponding 
compensator can be derived to minimize the error prob- 
ability of a PSK signal with nonlinear phase noise. 

Although very popular, MMSE compensator does not 
minimize the error probability after the compensator. To 
minimize the error probability, the optimal maximum a 
posteriori probability (MAP) detector |4J, Sec. 5.8] must 
be used. It is theoretically very important to find the op- 
timal compensator, possible by any mean and regardless 
of complexity or practicality, to combat nonlinear phase 
noise. An application of the optimal MAP compensator 
is to verify the optimality of a practical compensator. In 
order to find the optimal nonlinear MAP compensator, 
this paper first derives the joint distribution of the re- 
ceived amplitude and phase. The optimal nonlinear MAP 
compensator is then derived for PSK signal with nonlin- 
ear phase noise. The error probability of both nonlinear 
MAP and MMSE compensators is calculated using the 
joint distribution of received amplitude and phase. 

Later parts of this paper are organized as following: 
Sec. ITT1 builds the mathematical model of the nonlinear 
phase noise and derives the joint characteristic function 
of the normalized nonlinear phase noise and the elec- 
tric field with nonlinear phase noise. Sec. IIIII gives the 
marginal p.d.f of nonlinear phase noise by inverse Fourier 
transform. Sec. IIVI obtains the joint characteristic func- 
tions of the nonlinear phase noise with received inten- 
sity and/or the phase of amplifier noise. Using the joint 
characteristic function of nonlinear phase noise with the 



phase of amplifier noise, Sec. calculates the exact er- 
ror probability of PSK and DPSK signals with nonlinear 
phase noise. An approximation is also presented based 
on the assumption that the nonlinear phase noise is inde- 
pendent of the phase of amplifier noise. Using the joint 
characteristic function of nonlinear phase noise with re- 
ceived intensity, Sec. I VII provides the optimal linear com- 
pensators to compensate the nonlinear phase noise using 
received intensity. Using the joint characteristic function 
of nonlinear phase noise with both received intensity and 
phase of amplifier noise, the exact error probability of 
PSK and DPSK signals with linearly compensated non- 
linear phase is derived. Sec. IVIII discusses the nonlinear 
compensator for nonlinear phase noise to minimize either 
the error probability or the variance of residual nonlinear 
phase noise. Finally, Sec. IVIIII is the conclusion of this 
article. 



II. JOINT STATISTICS OF NONLINEAR 
PHASE NOISE AND ELECTRIC FIELD 

This section provides the joint characteristic function 
of nonlinear phase noise and the electric field without 
nonlinear phase noise. Both the nonlinear phase noise 
and the electric field are first normalized and represented 
as the summation of infinite number of independently dis- 
tributed random variables. The joint characteristic func- 
tion of nonlinear phase noise and electric field is the prod- 
uct of the corresponding joint characteristic functions of 
those random variables. After some algebraic simplifica- 
tions, the joint characteristic function has a simple ex- 
pression. 

A. Normalization of Nonlinear Phase Noise 

In a lightwave system, nonlinear phase noise is induced 
by the interaction of fiber Kerr effect and optical ampli- 
fier noise In this article, nonlinear phase noise is 
induced by self-phase modulation through the amplifier 
noise in the same polarization as the signal and within 
an optical bandwidth matched to the signal. The phase 
noise induced by cross-phase modulation from amplifier 
noise outside that optical bandwidth is ignored for sim- 
plicity. The amplifier noise from the orthogonal polar- 
ization is also ignored for simplicity. As shown later, we 
can include the phase noise from cross-phase modulation 
or orthogonal polarization by simple modification. 

For an TV-span fiber system, the overall nonlinear phase 
noise is H III S El 

0nl = lL c ft [\E + ni\ 2 + \Eq + fix + n 2 \ 2 H 

+ \E + + ■ ■ ■ + n N \ 2 }, (1) 

where Eq is a two-dimensional vector as the baseband 
representation of the transmitted electric field, ftk,k — 
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1, . . . , N, are independent identically distributed (i.i.d.) 
zero-mean circular Gaussian random vectors as the opti- 
cal amplifier noise introduced into the system at the k th 
fiber span, 7L c g is the product of fiber nonlinear coeffi- 
cient and the effective fiber length per span. In , both 
electric field of Eq and amplifier noises of can also be 
represented as a complex number. 

Figs-Hshow the simulated distribution of the received 
electric field including the contribution from nonlinear 
phase noise. The mean nonlinear phase shifts < $nl > 
are 1 and 2 rad for Figs. and^J}, respectively. The 
mean nonlinear phase shift of < $nl >= 1 rad corre- 
sponds to the limitation estimated by The mean 
nonlinear phase shift of <$nl>= 2 rad corresponds to 
the limitation given by 35] when the standard deviation 
of nonlinear phase noise is halved using a linear compen- 
sator. The limitation of <$nl>= 2 rad may be inferred 
from (allp. 

Figs. are plotted for the case that the signal-to- noise 
ratio (SNR) p s = 18 (12.6 dB), corresponding to an er- 
ror probability of 1CP 9 if the amplifier noise is the sole 
impairment. The number of spans is N = 32. The trans- 
mitted signal is E = (±|_E |,0) for binary PSK signal. 
The distribution of Figs. ^ nas 5000 points for different 
noise combinations. In practice, the signal distribution 
of Figs. can be measured using an optical phase- locked 
loop (see Fig. 5 of 45]). Note that although the optical 
phase-locked loop actually tracks out the mean nonlinear 
phase shift of <$nl>, nonzero values of <<&nl> have 
been preserved in plotting Figs-^to better illustrate the 
nonlinear phase noise. 

With large number of fiber spans, the summation of 
can be replaced by integration as [33, |38j 



Comparing the phase noise of and J3J), the normalized 
nonlinear phase noise of © is scaled by (f> = Lct^nl/k, 
t = z/L is the normalized distance, b(t) = S(tE)j o s j \f~L 
is the normalized amplifier noise, £o = Eq/ 'a s /y/L is 
the normalized transmitted vector, and the normalized 
electric field of e*jv is scaled by the inverse of the noise 
variance. The SNR of the signal is p s — |£o| 2 = 

\E \ 2 /(La 2 ) = \E Q \ 2 /(Na 2 ASE ).^ 

In 0, the normalized electric field e/v is the normal- 
ized received electric field without nonlinear phase noise. 
The actual normalized received electric field, correspond- 
ing to Fig. ^ is e r = ejv exp(— j(f>). The actual normal- 
ized received electric field has the same intensity as that 
of the normalized electric field e/v, i.e., \e r \ 2 = |eV| 2 . The 
values of y — |e*iv | 2 and r = | ejv | are called normalized 
received intensity and amplitude, respectively. 

B. Series Expansion 

The Brownian motion of b(t) can be expanded using 
the standard Karhunen-Loeve expansion of 0, Sec. 10- 
6] 



b(t) = ^dkXkipkit), (5) 
fc=i 

where Xk are i.i.d. two-dimensional circular Gaussian 
random variables with zero mean and unity variance 
of -E{|xfc| 2 } = 1, the eigenvalues and eigenfunctions of 
<*i < t < 1 are [H| HI p. 305] 



> NL = k I \E + S(z)\ dz, 
lo 



(2) 



where L is the overall fiber length, k = N-fL c s/L is 
the average nonlinear coefficient per unit length, and 
S(z) is a zero-mean two-dimensional Brownian motion of 
E{S(zi) ■ S(z2)} — er 2 min(zi, 22), where • denotes inner 
product of two vectors. The variance of a 2 = Naj^^/L 
is the noise variance per unit length where -E{|n/c| 2 } — 



'ASE 



,k = l, 



, N is noise variance per amplifier per po- 



larization in the optical bandwidth matched to the signal. 

In this article, we investigate the joint statistical prop- 
erties of the normalized electric field and normalized non- 
linear phase noise 



4 = Co + b(l), 4> ■ 



\Zo + Ht)\ 2 dt, 



(3) 



where b(t) is a two-dimensional Brownian motion with 
an autocorrelation function of 



R b (t, s) = E{b(s) ■ b(t)} = min(t, s). 



(4) 



07c 



(2k- 1)tt 



(2k- 1)tt 



(G) 



Substitute JSJ with into the normalized phase of 
©, because J Q sin(t/ak)dt = at, we get 



k=l k=l 

Because Y%Li a l = \ [ see Sec - 0.234]], we get 



(7) 



\ " 2 



ilv^o 



Xk\ 



(8) 



k=l 



The random variable Iv^Co + Xk\ 2 is a noncentral chi- 
square (x 2 ) random variable with two degrees of freedom 
with a noncentrality parameter of 2p s and a variance pa- 
rameter of \ [ill pp. 41-46]. The normalized nonlinear 
phase noise is the summation of infinitely many indepen- 
dently distributed noncentral x 2 -random variables with 
two degrees of freedom with noncentrality parameters of 
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(a) <$nl>= 1 rad 



(b) <$nl>= 2 rad 



FIG. 1: Simulated distribution of the received electric field for mean nonlinear phase shift of (a) <<E , nl> = 1 rad and (b) 
<$nl>= 2 rad. 



2a\p s and variance parameters of \<j\- The mean and 
standard deviation (STD) of the random variables are 
both proportional to the square of the reciprocal of all 
odd natural numbers. 

Using the series expansion of JHJ), the normalized elec- 
tric field is 



E- 



{exp jV|\/2|o + x k \ 2 + juj ■ (V2£ + x k ) j 



1 - ju 



— exp 



' 2j^ \ 2 + V2j^-iu~\\Luf 
1 - ju 



= Co + V2^(-l) fe + 1 



(9) 



fc=i 



Using HI Sec. 0.232], we get E^-l^+Vfc = \ and 



(-N 



= V2j2(-l) k+1 a k (V2£ - 



Xk) 



(10) 



k=l 



In the above expression, if uj = 0, the characteristic func- 
tion of |-\/2£o + x k \ 2 is 



(12) 



for a noncentral ^-distribution with mean and variance 
of 2p s + 1 and 4p s + 1, respectively 0> p. 42]. 
The joint characteristic function of g, is 



The normalized electric field of 110(1 is a two-dimensional 
Gaussian-distributed random variable having a mean of 
of £o an d variance of \. 

The series expansion of (jHJ and ((10(1 can be used to 
derived the joint characteristic function of the normalized 
electric field of ejv and nonlinear phase noise of in 



C. Joint Characteristic Function 

The joint characteristic function of the normalized non- 
linear phase noise and the electric field of J3J is 



x exp 



oo 



fc=i 



2jv\U°l + 2j(-l) fe+ V fe fo ■ £> - \al\ujf 



1 - j'vffjj 



(13) 



as the product of the joint characteristic function of the 
corresponding independently distributed random vari- 
ables in the series expansion of © and ((10(1 . 

Using the expressions of HI Sees. 1.431, 1.421, 1.422] 



e n ( v ^) =E{exp(ju(l> + juj-eN)}. (11) 



This joint characteristic function was derived by jj, |49j 
based on the method of [5(|. For completeness, a brief 
derivation is provided here using a significantly different 
method to eliminate some minor errors in [j, |4jJ . 
First of all, we have 



cos x 



TTX 



n • 



fc=i 



4a; 2 



(2k - 1) 2 7T 2 



4cc x - 

<7T ^-^ 



1 



7T ^ (2k 

k=l v 



4 °^ (_i)fc+i(2fc- 1) 
tt^ (2/s-l) 2 -x 2 : 
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the characteristic function of 1|13[1 can be simplified to 



exp 



\Zo\WP 



+j sec(Vj^)£o ■ u 



tany jv 



(14) 



The trigonometric function with complex argument is 
calculated by, for example, 



scc v jr) = (o M /^cosh J V - - jsin W^sinh J V -. 



(sec yfjv} 



The p.d.f. of p $ g (0, z) is the inverse Fourier trans- 
form of the characteristic function g (v 1 w) of (|14p. 
So far, there is no analytical expression for the p.d.f. of 

It is also obvious that 



4 



(15) 



is the characteristic function of a two-dimensional Gaus- 
sian distribution [Zsl pp. 48-51] for the normalized elec- 
tric field of (|TU|) 

In the field of lightwave communications, the approach 
here to derive the joint characteristic function of nor- 
malized nonlinear phase noise and electric field is simi- 
lar to that of |5lJ to find the joint characteristic func- 
tion for polarization- mode dispersion [H^ , or that of j5^| 
for filtered phase noise. Another approach is to solve 
the Fokker- Planck equation of the corresponding diffu- 
sion process psij . 



III. THE PROBABILITY DENSITY OF 
NONLINEAR PHASE NOISE 

The characteristic function of the normalized nonlinear 
phase noise is vP^, g (v, 0) or |88l ] 



^^(v) = seci/jVexp p s \J jvt&nyj jv 



(16) 



From the characteristic function of (|16[) . the mean nor- 
malized nonlinear phase shift is 



<$>= -j — 
av 



1 

= Ps + - 2 - 



(17) 



Note that the differentiation or partial differentiation op- 
eration can be handled by most symbolic mathematical 
software. The scaling from normalized nonlinear phase 
noise to the nonlinear phase noise of J2J is 
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FIG. 2: The p.d.f. of the normalized nonlinear phase noise 
for SNR of p s = 11, 18, and 25. 



5NL = 



<$NL> 



Ps+ 2 

The second moment of the nonlinear phase noise is 



(18) 



<$ 2 >= - ^(zy) 



2 



that gives the variance of normalized phase noise as 



(19) 



(20) 



The p.d.f. of the normalized nonlinear phase noise of 
© can be calculated by taking the inverse Fourier trans- 
form of the characteristic function 1)16(1 . Fig.|2shows the 
p.d.f. of the normalized nonlinear phase noise for three 
different SNR of p s = 11, 18, and 25 (10.4, 12.6 and 14.0 
dB), corresponding to about an error probability of 10~ 6 , 
10 -9 , and 10 -12 , respectively, when amplifier noise is the 
sole impairment. Fig. [21 shows the p.d.f. using the exact 
characteristic function l|16|) . and the Gaussian approxi- 
mation with mean and variance of < $ > (j!7[l and a\ 
H20JI . From Fig. [3 the Gaussian distribution is not a 
good model for nonlinear phase noise. 

The p.d.f. for finite number of fiber spans was derived 
base on the orthogonalization of the nonlinear phase 
noise of Q by the summation of N independently dis- 
tributed random variables (37|. Fig. [3] shows a compari- 
son of the p.d.f. for N = 4, 8, 16, 32, and 64 of fiber spans 
[3?| | with the asymptotic case of l)16[l. Using the SNR of 
p s = 18 (12.6 dB), Fig. O is plotted in logarithmic scale 
to show the difference in the tail. Fig. also provides an 
inset in linear scale of the same p.d.f. to show the dif- 
ference around the mean. The asymptotic p.d.f. of i|16|) 
with distributed noise has the smallest spread in the tail 
as compared with the p.d.f. with TV discrete noise sources. 
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Nonlinear Phase Noise, <]>/(p + 1/2) 

FIG. 3: The asymptotic p.d.f. of normalized nonlinear phase 
noise of <j> as compared with the p.d.f. of N = 4, 8, 16, 32, and 
64 fiber spans. The p.d.f. in linear scale is shown in the inset. 

The asymptotic p.d.f. is very accurate for N > 32 fiber 
spans. 

Comparing Figs. [21 and [3] when the p.d.f. is plotted in 
linear scale, the difference between the actual and Gaus- 
sian approximation seems small as compared with the 
same p.d.f. plots in logarithmic scale. In linear scale, the 
p.d.f. also seems more symmetric in Fig. [5] or the inset of 
Fig. E| W hen phase noise is very large or plots in linear 
scale |55l IH^ . the Gaussian approximation seems more 
valid. 

As discussed earlier, the effects of amplifier noise out- 
side the signal bandwidth and the amplifier noise from 
orthogonal polarization are all ignored for simplicity. If 
the nonlinear phase noise induced from those amplifier 
noises is included, based on the simple reasoning of [13, 
the marginal characteristic function of the normalized 
nonlinear phase noise of (|16|) becomes 



and noise. In later part of this article, the characteristic 
function of (|16[) can be changed to (|2H if necessary. 

The characteristic function of 121(1 assumes a disper- 
sionless fiber. With fiber dispersion, due to walk-off 
effect, the nonlinear phase noise caused by cross-phase 
modulation should approximately have Gaussian distri- 
bution. Method similar to |^ E3 can be used to find 
the variance of the nonlinear phase noise due to cross- 
phase modulation in dispersive fiber. For either PSK 
and DPSK signals, the signal induced nonlinear phase 
noise by cross-phase modulation should be very small. 
The power spectral density of signal and noise can be 
first derived, multmlied by the transfer function due to 
walk-off from [HsL l59j | and integrated over all frequency 
gives the variance of phase noise. 

For DPSK signal, the phase noises in adjacent sym- 
bols are correlated to each other [(5(j ■ The characteris- 
tic function of the differential phase due to cross-phase 
modulation can be found using the power spectral den- 
sity of ,58;], taking the inverse Fourier transform to get 
the autocorrelation function, and getting the correlation 
coefficient as the autocorrelation with a time difference 
of the symbol interval. The characteristic function of the 
differential phase decreases by the correlation coefficient. 

Similar to Q, OH , all the derivations here assume 
non-return-to-zero (NRZ) pulses (or continuous- wave sig- 
nal) but most experiments BIlHlISHHIISIIJEl 
HI, H3 use return-to-zero (RZ) pulses. For flat-top 
RZ pulse, the mean nonlinear phase shift of < <I>nl > 
should be the mean nonlinear phase shift when the peak 
amplitude is transmitted. Usually, <$nl> is increased 
by the inverse of the duty cycle. However, for soliton 
and dispersion-mana ged soliton , based on soliton per- 
turbation [6ll l62l I63ll64l ] or variational principle 0, y(| , 
the mean nonlinear phase shift of < < I ) nl> is reduced by 
a factor of 2 when dispersion and self-phase modulation 
balance each other Hal. 



sec ^ 



(^0 



exp 



(21) 



IV. SOME JOINT CHARACTERISTIC 
FUNCTIONS 



where m is product of the ratio of the amplifier noise 
bandwidth to the signal bandwidth and the number of 
polarizations. If only the amplifier noise from orthog- 
onal polarization matched to signal bandwidth is also 
considered, m = 2 for two polarizations gives the char- 
actersitic function of (|16fl . With cross-phase modulation 
induced nonlinear phase noise, the mean and variance of 
the nonlinear phase noise increase slightly to p s + \m 
and |p s + |m, respectively. The nonlinear phase noise is 
induced mainly by the beating of the signal and amplifier 
noise from the same polarization as the signal, similar to 
the case of signal-spontaneous beat noise in an ampli- 
fied receiver. For high SNR of p s , it is obvious that the 
signal-amplifier noise beating is the major contribution to 
nonlinear phase noise. The parameter of m can equal to 
1 for the case if the amplifier noise from another dimen- 
sion is ignored by confining to single-dimensional signal 



From the characteristic function l|14|) . we can take the 
inverse Fourier transform with respect to Co and get 

^{*^(^)}=^K%^^}' (22) 

where J-Z 1 denotes the inverse Fourier transform with 
respect to to, and T$ denotes the Fourier transform with 
respect to <j>. The characteristic function of 114fl can be 
rewritten as 



(^) = **(^)exp 



\lo\ 2 tany^ 



+jscc( v /j^)Co ■ <3 



(23) 
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where ^(y) is the marginal characteristic function of 
nonlinear phase noise from Ijltjfl . The inverse Fourier 
transform is 



i&<i>(i>) 
WT eXP 



(24) 



1 - 2jua\ 



exp 



1 - 2jW* 



(29) 



where cr 2 = ^ taa{-\fjv) / an d ^ = sec (v7^)Co- Both 
a 2 and are complex numbers and can be considered 
as the angular frequency depending variance and mean, 
respectively. 



A. Joint Characteristic Function of Nonlinear 
Phase Noise and Received Intensity 

Using the partial p.d.f. and characteristic function of 
l[24jl. change the random variable from rectangular co- 
ordinate of z = (21,22) to polar coordinate of z = 
(r cos 9, r sin 9), we get 



cosyji/ — juj- 



x exp ps^J jv\m\\J jv 



cos 2 vV - 3^ tM 



(30) 



The joint characteristic function of l|3(J|l can be used to 
study the compensation of nonlinear phase noise using 
received intensity [33l 03 H3| • 



lira?, 



x exp 



r 2 + |e,| 2 -2r|^|cos( 



2a, 2 , 



■ (25) 



where 9q is the angle of the transmitted vector £0 and 
|£„| = sec(v^)|Co|- The random variable of 0„ is called 
the phase of amplifier noise because it is solely con- 
tributed from amplifier noise. 

Taking the integration over 9 and changing the random 
variable to the received intensity of y = r 2 , we get 



~2aJ 



■ exp 



y + \U 

2a 2 



r 



(26) 



where /&(•) is the A: th -order modified Bessel function of 
the first kind. The p.d.f. of the received intensity of 



= exp (-y - p s ) I {2^/yp~ s 



(27) 



is a non-central x 2_ P-d.f. with two degrees of freedom 
with a noncentrality parameter of p s and variance pa- 
rameter of i [i^. pp. 41-44]. With a change of random 
variable of y = r 2 , the received amplitude has a Rice 
distribution of [H, pp. 46-47] 

p R (r) = 2rexp [-{r 2 + p.)] J (2ry^). (28) 

Taking a Fourier transform of l|26|l . the joint charac- 
teristic function of nonlinear phase noise and received 
intensity is 



B. Joint Characteristic Function of Nonlinear 
Phase Noise and the Phase of Amplifier Noise 

Using the characteristic function of (|25|l . take the in- 
tegration over the received amplitude r, we get 



x / exp 
'0 



<£$(z/) 
27ra 2 

r 2 + |^| 2 -2r|e;|cos( 



-0a) 



2a. 



rdr, 



or 



^T 1 {**,e R } = *«(f) 




<erfc[-^cos(9-9 )} >, (31) 



where 



2^ 



2a 2 sin (2^7) ' 



(32) 



can be interpreted as the angular frequency depending 
SNR. 

Taking the Fourier transform of (|31|l . from [&5l Sec. 
9.2-2], the characteristic function of ^<i>.e n is 
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x E e ' 

m=0 



iv „ ffn \ „ /m \ 
^! r (2 +1 )^(T ;m + 1; ~>) 



(e 



2^.7(771+^) 



l)e- 



2jn(m + w) 



2jTt(u> — m) 



(33) 



where T(-) is the Gamma function, i-Fi(a; 6; ■) is the con- 
fluent hypergeometric function of the first kind with pa- 
rameters of a and b, and e m = 1 if m = 1, e m = 2 if 
m > 1. 

Within the summation of the joint characteristic func- 
tion of l|33l) . if a; = m as an integer, only one term in the 
summation is non-zero. We have 



<i>.0 



(v, to) 



(m 

XiFiy—;m+l;-j v 
-¥*(i/) 7 £exp (-^) 



/TO \ 

t 2 + 



o jm0 o 



TO > 0, 



(34) 



and *$ ie „(i/,-m) = **,e„(^, TO)e~ 2ime °. Using H3, 
Sec. 9.212, Sec. 9.238], the conversion from hyperge- 
ometric function to Bessel functions in l|34|l is used in 
0, EE EE EH • The simple expression for \&$,e„ (f, to) is 
very helpful to derive the p.d.f. of the signal phase. The 
coefficients of <|34[) are also the Fourier series coefficients 
of the expression of (|31|l expanded over the phase 9 in 
the range of [— 7T, 7r). 



< / exp 
Jo 



2a 



■*m 1 o 



e^dy, 



to > 0, 



and f#,m(i/,w,-m) = *$,y,e„ (^, m)e~ 2:, ' m80 . 
Using g3 Sec. 6.614, Sec. 9.220], we get 



^ , 4>,y,e rl (j / , to) = \f'$(;/)exp 



2(7?, 



r(i m + i) 



m! 



(l-2j,Wg)t|^| V2 

TO > 0, 



(36) 



where 



7«V 



161 



1 - 2jW* 2a 2 



2jV 



[V^ ~ i w tan (\^)] sin ( 2 v^) 
Using 7^ defined by l|32")l . we can rewitc as 



P S - (37) 



.)„(//. a,-. /» j = *$(;/)- 
1 

X ■ 



-T-+j' roe °r(iTO + 1) 



: 2^ lFl (^ + l ;m+ l ;7 _). (38 ) 
7^ V 2 J 

With w = in (JSHJl, using jH Sec. 9.212], the joint 
Fourier coefficients of nonlinear phase noise and the phase 
of amplifier noise are 



C. Joint Characteristic Functions of Nonlinear 
Phase Noise, Received Intensity and the Phase of 
Amplifier Noise 

Here, we derive the joint characteristic functions of 
^<s>,Y,e n - Similar to l|34|) . corresponding to the Fourier 
coefficients, only the characteristic function at integer 
"angular frequency" of n is interested. With m as an 
non-negative integer, using 0> Sec. 8.431] and lt2"B|) with 
y = r 2 , we get 



x exp 



2al 



T ( Vy&l 



, to > 0, (35) 



Taking the Fourier transform of H35|l . we get 



**,y,e„(f, v,m) 



2^H 



^*.e„(^j TO ) = V E'$(^)- 



-T"+^ meo r(im + l) ™ 



-7r/ 



/TO \ 

X1-F1 ^— + 1;to + 1;j„J 

= — ^ r (y + 1 ) lFl (y ;m + 1; ->)' 

the same as (|34|) . 

If to = in using the relationship of 1.F1 (1; 1; z) = 
e z [471 Sec. 9.215], the joint characteristic function of 
nonlinear phase noise and the received intensity is 



1 - 2^a 2 



Iv 

exp 



1 - 2]wal 



(39) 



also the same as (|30|l . 

To simplify l|38|) using the Bessel functions, from |47l 
Sec. 9.212, Sec. 9.238] and similar to [13, ES |s3, El , we 

get 



9 



e„ (f, w, ™) = — - — exp 



7i/,u 
2 



(40) 



Although both joint characteristic functions l|30|) and 
(|34|l can be derived based on the joint characteristic func- 
tion H38|) or 1)41) |). they are derived seperately in early 
parts of this section for simplicity. 



V. ERROR PROBABILITY OF 
PHASE-MODULATED SIGNALS 



Binary DPSK signaling with interferometer based 
direct-detection receiver has renewed interests recently 
El El Elll El El- This section studies the impact 
of nonlinear phase noise to binary PSK and DPSK sig- 
nals. In order to derive the error probability, the p.d.f. of 
the received phase, that is the summation of both nonlin- 
ear phase noise and the phase of amplifier noise, is first 
derived analytically as a Fourier series. Taking into ac- 
count the dependence between the nonlinear phase noise 
and the phase of amplifier noise, the error probability is 
calculated using the Fourier coefficients. 

If the nonlinear phase noise is assumed to be Gaussian 
distributed, the error probability is the same as that of 
[4l| with laser phase noise. Because laser phase noise is 
a Brownian motion, the phase noise difference between 
two consecutive symbols is Gaussian distributed. 

The optimal operating point of the system is estimated 
by based on the insight that the variance of linear 
and nonlinear phase noise should be approximately the 
same. With the exact error probability, the system can be 
optimized rigorously by the condition that the increase in 
SNR penalty is less than the increase of launched power. 



A. Phase Distribution 

Without loss of generality, in this section, the nor- 
malized transmitted electric field is assumed to be £o = 
(y/pl, 0) when 9q = 0. With nonlinear phase noise, re- 
ceived by an optical phase- locked loop 0,Ei|, the overall 
received phase is 



*r = e„ - $ NL = 9„ - (41) 

Ps + 2 

where p s + \ is the mean normalized nonlinear phase shift 

The received phase is confined to the range of [— it, +tt) . 
The p.d.f. of the received phase is a periodic function 
with a period of 2ir. If the characteristic function of the 
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FIG. 4: The p.d.f. of the received phase p& r .(0+ <$nl>) in 
logarithmic scale. The inset is the p.d.f. of p<s> T (6) in linear 
scale. 



received phase is 4'$ r (^), the p.d.f. of the received phase 
has a Fourier series expansion of 



-t-oo 



P<$> T 



r (0) = ir Y, ** r (m)exp(jm0). (42) 



m— — oo 



Because the characteristic function has the property of 
**r(- v ) = we g et 



(«) = | + -E S ^ M exp(jmf?)} , (43) 

J it tt — ' 



Pt>r 



2ir 7r 



where denotes the real part of a complex number. 

Using the joint characteristic function of 4'$ l o rl (i', m) 
(|34p. from the received phase of Q41f) . the Fourier series 
coefficients are 



*$ r (m) = * 



<$NL> 



(44) 



Fig.0]shows the p.d.f. of the received phase (|43|l with 
mean nonlinear phase shift of <$nl>= 0,0.5,1.0,1.5, 
and 2.0 rad. Shifted by the mean nonlinear phase shift 
< < I , nl>, the p.d.f. is plotted in logarithmic scale to show 
the difference in the tail. Not shifted by < $nl > , the 
same p.d.f. is plotted in linear scale in the inset. Fig.0] 
is plotted for the case that the SNR is equal to p s = 18 
(12.6 dB), corresponding to an error probability of 10 -9 if 
amplifier noise is the sole impairment. Without nonlinear 
pha se noise <$nl>= 0, the p.d.f. is the same as that in 
|48l Sec. 5.2.7] and symmetrical with respect to the zero 
phase. 

From Fig. when the p.d.f. is broadened by the non- 
linear phase noise, the broadening is not symmetrical 
with respect to the mean nonlinear phase shift <$nl>- 
With small mean nonlinear phase shift of <<! ) nl>= 0.5 
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rad, the received phase spreads further in the positive 
phase than the negative phase. With large mean non- 
linear phase shift of <$nl>= 2 rad, the received phase 
spreads further in the negative phase than the positive 
phase. The difference in the spreading for small and large 
mean nonlinear phase shift is due to the dependence be- 
tween nonlinear phase noise and the phase of amplifier 
noise. As shown in Fig. [21 after normalization, the p.d.f. 
of nonlinear phase noise depends solely on the SNR. If 
nonlinear phase noise is independent of the phase of am- 
plifier noise, the spreading of the received phase noise is 
independent of the mean nonlinear phase shift <<!>nl>- 



B. Error Probability of PSK Signals 

If the p.d.f. of H43|) were symmetrical with respect to 
the mean nonlinear phase shift < $nl >, the decision 
region would center at the mean nonlinear phase shift 
<$nl> and the decision angle for binary PSK system 
should be ±^7r— <$nl>- From Fig.0] because the p.d.f. 
is not symmetrical with respect to the mean nonlinear 
phase shift < $nl > , assume that the decision angle is 
±^7r— C with the center phase of 9 C , the error probability 
is 



Pe.PSK = 1 



P* r (fi)M, 



(45) 



or 



1 1 S 2sin(im7r) , , g , 

2 7r m 

rn—l 



After some simplifications, we get 



(46) 



+ 00 



(47) 

From the characteristic function of Q34JI. the coeffi- 
cients for the error probability (|47|l are 



*1 (2fc + l) 



(2k + 1) <$ NL > 



J s T 2 



k > 0, (48) 



where, from l|32|l . 



At — 
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"j(2fe + l)<*NL>" 


2 










sin|2 


"j(2fc+l)<* NL >" 


1 ^ 

"} 







(49) 



are equivalent to the angular frequency depending SNR 
parameters. 

Note that the exact error probability (|47[) is very sim- 
ilar to that in However, the error probability of eq. 
(71) of 4] is for PSK instead of DPSK signal. This will 
be more clear in later parts of this paper. The major dif- 
ference between the exact error probability (|47|l and that 
in |4| is the observation that the center phase is not equal 
to the mean nonlinear phase shift. From (|16fl . the shape 
of the p.d.f. of nonlinear phase noise depends solely on 
the signal SNR. 

In the coefficients of (|48|l . the complex coefficients of 
(|49|l are equivalent to the angular frequency depending 
SNR parameters. Bessel functions with complex argu- 
ment are well-defined [7(| • 

The coefficients of (I48|) have a very complicated ex- 
pression because of the dependence between the phase of 
amplifier noise and the nonlinear phase noise. Because 
E{<d n &Nh} = 0, the phase of amplifier noise and the 
nonlinear phase noise are uncorrelated with each other. 
However, uncorrelated is not equivalent to independence 
for non-Gaussian random variables. 

If the nonlinear phase noise is assumed to be indepen- 
dent to the phase of amplifier noise |8|], similar to the 
approaches of [iol El) in which the extra phase noise is 
independent of the signal phase, the error probability can 
be approximated as 



Pe,PSK 



1 

2 

x3?<j 



* (-1) 



k r 



7T ^ 2k 

k=0 



1 



Ik 



4+i(y) 



(2k + 1) <$ NL > 



-j(2k+l)9 c 



(50) 



In , the center phase of 9 C of l|50|) is assumed to be the 
mean nonlinear phase shift <$nl>- 

Fig. shows the exact |4*7)l and approximated f5H|l er- 
ror probabilities as a function of SNR p s . Fig. [S] also 
plots the error probability without nonlinear phase noise 



of p e ,psK = ^erfcy^ |42|, Sec. 5.2.7]. Fig. plots the 
error probability for both the center phase equal to the 
mean nonlinear phase shift 9 C =<"I > nl> (empty symbol) 
Q and optimized to minimize the error probability (solid 
symbol) . The approximated error probability in Fig. 
with 6 C =<$nl> is the same as that in 8] but calculated 
by a simple formula of l|50l) . From Fig. \5\ with optimized 
center phase, the approximated error probability (f 501) al- 
ways underestimates the error probability. 

Fig. El shows the SNR penalty of PSK signal for an er- 
ror probability of 10 -9 calculated by the exact (|47|l and 
approximated l|50|) error probability formulae. Fig. El is 
plotted for both cases of the center phase equal to the 
mean nonlinear phase shift 9 C =<$nl> or optimized to 
minimize the error probability. The corresponding opti- 
mal center phase is shown in Fig. [7| When the center 
phase is equal to the mean nonlinear phase shift, the re- 
sults using the exact error probability (|47|l should be the 
similar to that of . When the center phase is equal to 
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FIG. 5: The error probability of PSK signal p e ,psK as a func- 
tion of SNR p a . 
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FIG. 6: The SNR penalty of PSK signal as a function of mean 
nonlinear phase shift <$nl>- 



the mean nonlinear phase shift C =<<I>nl>, the SNR 
penalty given by the approximated error probability l|5U|) 
is the same as that in |8| but calculated by a simple for- 
mula. 

The discrepancy between the exact and approximated 
error probability is smaller for small and large nonlinear 
phase shift <$nl>- With the optimal center phase, the 
largest discrepancy between the exact and approximated 
SNR penalty is about 0.49 dB at a mean nonlinear phase 
shift of <<I>nl> around 1.25 rad. When the center phase 
is equal to the mean nonlinear phase shift 9 C =<$nl>, 
the largest discrepancy between the exact and approxi- 
mated SNR penalty is about 0.6 dB at a mean nonlinear 
phase shift of <$nl> around 0.75 rad. For PSK signal, 
the approximated error probability (|50|l may not accu- 
rate enough for practical applications. 



Using the exact error probability (|47|l with optimal 
center phase, the mean nonlinear phase shift < $nl > 
must be less than 1 rad for a SNR penalty less than 1 
dB. The optimal operating level is that the increase of 
mean nonlinear phase shift, proportional to the increase 
of launched power and SNR, does not decrease the system 
performance. In Fig. the optimal operation point can 
be found by 



dp s 



d <$NL> 



(51) 



when both the required SNR p s and mean nonlinear 
phase shift <$nl> ar e expressed in decibel unit. The 
optimal operating level is for the mean nonlinear phase 
noise <$nl> of about 1.25 rad, close to the estimation 
of when the center phase is assumed to be <<Fnl>- 

From the optimal center phase of Fig. with the ex- 
act error probability l|47|l . the optimal center phase is 
less than the mean nonlinear phase shift < < &nl> when 
the mean nonlinear phase shift is less than about 1.25 
rad. At small mean nonlinear phase shift, from Fig. 0] 
the p.d.f. of the received phase spreads further to posi- 
tive phase such that the optimal center phase is smaller 
that the mean nonlinear phase shift <$nl>- At large 
mean nonlinear phase shift <<&nl>, the received phase 
is dominated by the nonlinear phase noise. Because the 
p.d.f. of nonlinear phase noise spreads further to the 
negative phase as from Fig. |2 the optimal center phase 
is larger than the mean nonlinear phase shift for large 
mean nonlinear phase shift. For the same reason, when 
the nonlinear phase noise is assumed to be independent of 
the phase of amplifier noise, the optimal center phase is 
always larger than the mean nonlinear phase shift. From 
Fig. 13 the approximated error probability (|50|l is not 
useful to find the optimal center phase. 

Comparing the exact l|47|l and approximated l|50|l er- 
ror probability, the approximated error probability l|50l) 
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FIG. 8: The direct-detection receiver of DPSK signal using 
an interferometer. 



is evaluated when the parameter of A& is approximated 
by the SNR p s . The parameters of A& are complex num- 
bers. Because |Afe| are always less than p s , with optimized 
center phase and from Figs. [3] and H3 the approximated 
error probability of H5()jl always gives an error probability 
smaller than the exact error probability 147(1 . 
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C. Error Probability of DPSK Signals 

Fig. |H1 shows the direct-detection receiver for DPSK 
signal. The DPSK receiver uses a Mach-Zehnder interfer- 
ometer in which the signal is splitted into two paths and 
combined with a path difference of a symbol time of T. In 
practice, the path difference r w T must be chosen such 
that expCiynr) = 1, where cu n is the angular frequency of 
the signal [63, Iz3, IzS (IS [zl, IzM (Z^l Ideally, the optical 
filter before the interferometer is assumed to be a match 
filter to the transmitted signal. Two balanced photode- 
tectors are used to receive the photocurrent. There is a 
low-pass filter to filter out the receiver noise. We assume 
that the low-pass filter has a wide bandwidth and does 
not distort the received signal. 

Optical amplified direct-detection DPSK receiver had 
been studied by [H HI El ^ 

The analysis here just 
takes into account the amplifier noise from the same po- 
larization as the signal [73 and can also be applied to 
heterodyne receiver [77| . 

The interferometer of Fig.[S]finds the differential phase 

of 



A$ r = $ r (i) - $ r (t- T) 

= B (t) - $ NL (t) - G n (t — T) + $ NL (< - T),(52) 

where $ r ( - )j 6 n ('), and <I>nl(') are the received phase, 
the phase of amplifier noise, and the nonlinear phase 
noise as a function of time, and T is the symbol interval. 
The phases at t and t — T are independent of each other 
but are identically distributed random variables similar 
to that of (|41|l . The differential phase of (|52|l assumes 
that the transmitted phases at t and t — T are the same. 

When two random variables are added (or subtracted) 
together, the sum has a characteristic function that is 
the product of the corresponding individual character- 
stic functions. The p.d.f. of the sum of the two random 
variables has Fourier series coefficients that are the prod- 
uct of the corresponding Fourier series coefficients. From 
(|43|1 . the p.d.f. of the differential phase l|52l) is 



FIG. 9: The p.d.f. of the differential received phase PA<s> r {0) 
in logarithmic scale. The inset is the same p.d.f. in linear 
scale. 



PA* r (0) = 7T + -Y l**,M| 2 cos(mfl). (53) 

Z7T 7T £ — ' 

m— 1 

As the difference of two i.i.d. random variables, with 
the same transmitted phase in two consecutive symbols, 
the p.d.f. of the differential phase A<j> r is symmetrical 
with respect to the zero phase. 

Fig. OH shows the p.d.f. of the differential received 
phase (|53|) with mean nonlinear phase shift of < ( &nl>= 
0, 0.5, 1.0, 1.5, and 2.0 rad. The p.d.f. is plotted in loga- 
rithmic scale to show the difference in the tail. The same 
p.d.f. is plotted in linear scale in the inset. Figgis plot- 
ted for the case that the SNR is equal to p s — 20 (13 dB), 
corresponding to an error probability of 10 -9 if amplifier 
noise is the sole impairment 77] . From Fig. |5J when the 
p.d.f. of differential phase is broadened by the nonlinear 
phase noise, the broadening is symmetrical with respect 
to the zero phase. 

Interferometer based receiver [S3, Iz3> Il3 gives an out- 
put proportional to cos(A<I> r ). The detector makes a de- 
cision on whether cos(A$ r ) is positive or negative that 
is equivalent to whether the differential phase A$ r is 
within or without the angle of ±5^. Similar to that for 
PSK signal (|47|) . the error probability for DPSK signal is 

1 +°° 

Pe.DPSK = 2 " " E WTl + 1)|2 • (M) 

where the coefficients of *&$ r (2fc + 1) are given by (|48|) . 

Similar to the approximation for PSK signal l|50|l . if 
the nonlinear phase noise is assumed to be independent 
to the phase of amplifier noise, the error probability of 
(|54|l can be approximated as 

1 p s e~P- ^ (_!)*= r (Pa ^ 
Pe.DPSK ~ g - — ^ ^ [h{ Y ) + h + l{ Y )_ 



13 




FIG. 10: The error probability of DPSK signal as a function 
ofSNRp s . 



(2k + 1) <$ NL > 



(55) 



Comparing the exact l|54|) and approximated (|55|l error 
probability, the approximated error probability l|55|) is 
evaluated when the parameter of is approximated by 
the SNR p s . Because |Afc| is always less than p s , the 
approximated error probability of Q55JI always gives an 
error probability smaller than the exact error probability 

(El- 
Fig, shows the exact (I54f) and approximated H55|) 
error probabilities as a function of SNR p s . Fig. I1UI also 
plots the error probability without nonlinear phase noise 
of p e , dpsk = ^exp(— p s ) 0j Sec. 5.2.8]. The approx- 
imated error probability in Fig. 1101 is the same as that 
in |8| but calculated by a simple formula of l|55|l . From 
Fig. the approximated error probability l|55(l always 
underestimates the error probability. 

Fig. El shows the SNR penalty of DPSK signal for 
an error probability of 10~ 9 calculated by the exact l|54|) 
and approximated (|55|l error probability formulae. The 
SNR penalty given by the approximated error probabil- 
ity is the same as that in Q but calculated by a simple 
formula (|55|l . The discrepancy between the exact and ap- 
proximated error probability is very small for small and 
large nonlinear phase shift <$nl>- The largest discrep- 
ancy between the exact and approximated SNR penalty 
is about 0.27 dB at a mean nonlinear phase shift <<I ) nl> 
of 0.53 rad. 

For a power penalty less than 1 dB, the mean nonlinear 
phase shift < $nl > must be less than 0.57 rad. The 
optimal level of the mean nonlinear phase shift <$nl> 
is about 1 rad such that the increase of power penalty 
is always less than the increase of mean nonlinear phase 
shift, similar to the estimation of as the limitation of 



the mean nonlinear phase shift. 

The error probabilities of Figs . 151 and 1101 are calculated 
using Matlab. The series summation of H47[) and (|54|) can 
be calculated to an error probability of 10~ 13 to 10~ 14 
with an accuracy of three to four significant digits. Sym- 
bolic mathematical software can provide better accuracy 
by using variable precision arithmetic in the calculation 
of low error probability. 



VI. LINEAR COMPENSATION OF 
NONLINEAR PHASE NOISE 



As shown in the helix shape scattergram of Figs. ^ 
nonlinear phase noise is correlated with the received in- 
tensity. The received intensity can be used to compensate 
for the nonlinear phase noise. Ideally, as shown in com- 
ing sections, the optimal compensator should minimize 
the error probability of the system after compensation. 
If the joint p.d.f. of the received phase and the received 
intensity of p$ ri y(0, y) is available, the optimal compen- 
sator or detector is given by the maximum a posteriori 
probability (MAP) criterion 0, Sec. 5.2] to minimize 
the error probability. For simplicity, linear compensator 
is discussed here first. The next section is about nonlin- 
ear compensator. 

In this section, the linear compensator is first opti- 
mized in term of the variance, or minimum mean-square 
error (MMSE) criterion, of the residual nonlinear phase 
noise. Afterward, the exact error probability with linear 
compensator is derived analytically. With a simple ex- 
pression to calculate the error probability, numerical op- 
timization is used to find the linear MAP compensator 
to minimize the error probability. Not for PSK signals, 
linear MMSE compensator performs close to linear MAP 
compensator for DPSK signals. 
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A. MMSE Linear Compensation 

The simplest method to compensate the nonlinear 
phase noise is to add a scaled received intensity into the 
received phase [H HUE El- The °P timal linear MMSE 
compensator minimizes the variance of the normalized 
residual phase noise of $ Q = $ — aR 2 = $ — aY . Using 
the joint characteristic function of (|30[) . the characteris- 
tic function for the normalized residual nonlinear phase 
noise is 



^4> Q ( u ) = ^<&,y{v, —olv). 



(56) 




-0.4 -0.2 0.2 0.4 0.6 0.8 1 1.2 
Residual Nonlinear Phase Noise, § /(p + 1/2) 



The mean of the normalized residual nonlinear phase 
noise is 



<$a> = ~j (f) 

dv 



(57) 



The variance of the normalized residual nonlinear phase 
noise is 



^2 



^ T , ^ 



i/=0 



= ^p s + i - 2 (p s + a + (2 Ps + 1) a 2 . (58) 

Solving da\ /da = 0, the optimal scale factor for linear 
compensator is 



1 Ps 



(59) 



In high SNR, a m i n — > t>- Other than the normalization, 
the optimal scale factor of l|59fl is the same as that in 
|35j. The approximation of a m i n — ► ^ was estimated by 
[34] though simulation. 

With the optimal scale factor of l|59fl . the mean and 
variance of the normalized residual nonlinear phase noise 
are 



,> 



1 pi + hs 



2 

a <S> a 



1 pi 



6 p s 



(60) 
(61) 



The mean of the residual nonlinear phase noise is about 
half the mean of the nonlinear phase noise of < $ > 
(|17|l . The variance of the residual nonlinear phase noise 
is about a quarter of that of the variance of the nonlinear 
phase noise of a\ (|2U|) . 



FIG. 12: The asymptotic p.d.f. of the residual nonlinear phase 
noise 4> a as compared with the p.d.f. of N = 4, 8, 16, 32, and 
64 fiber spans. The p.d.f. in linear scale is shown in the inset. 



After the linear compensation, the characteristic func- 
tion of the residual normalized nonlinear phase noise is 



(v) = *$,y V, -- 



V_Ps_ 

2 p s 



(62) 



The p.d.f. of the residual normalized nonlinear phase 
noise is the inverse Fourier transform of , J , $ ci (v). 

The p.d.f. of the residual nonlinear phase noise for fi- 
nite number of fiber spans was derived by modeling the 
residual nonlinear phase noise $ Q as the summation of N 
independently distributed random variables j3]j ■ Fig. IT21 
shows a comparison of the p.d.f. for N = 4, 8, 16, 32, and 
64 of fiber spans with the distributed case of i|tj2|l . 
The residual nonlinear phase noise is scaled by the mean 
normalized phase shift of p s + \ <(T7|) . Using a 

SNR of p s = 18, Fig.^]is plotted in logarithmic scale to 
show the difference in the tail. Fig. El also provides an 
inset in linear scale of the same p.d.f. to show the differ- 
ence around the mean. Like that of Fig. [21 the asymptotic 
p.d.f. for residual nonlinear phase noise of Fig. ^1 is also 
very accurate for N > 32 fiber spans. Unlike that of 
Fig. [31 the asymptotic p.d.f. for residual nonlinear phase 
noise of Fig. ^1 have slightly larger spread then that of 
the finite cases. The mean of the residual nonlinear phase 
noise is about 0.5(p s + \) , the same as that of (I6U|I . Com- 
paring Fig.[3Jand Fig. 1 121 with linear compensation, both 
the mean and STD of the residual nonlinear phase noise 
is about half of that of the nonlinear phase noise before 
compensation. 



Distribution of the Linearly Compensated 
Received Phase 



With nonlinear phase noise, received by an optical 
phase- locked loop [45L l69l | , the overall received phase is 
that of (|41|l . With linear compensation, the compensated 
received phase is 
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FIG. 13: The p.d.f. of the compensated received phase 
P*cm($+ < "^res >) in logarithmic scale. The inset is the 
p.d.f. of p* cm (#) in linear scale. 



the p.d.f. is plotted in logarithmic scale to show the 
difference in the tail. Not shifted by <$res>) the same 
p.d.f. is plotted in linear scale in the inset. Fig. 1131 is 
plotted for the case that the SNR is equal to p s = 18 
(12.6 dB), the same as that of Fig.0| 

From Fig. ^5] when the p.d.f. is broadened by the 
nonlinear phase noise, similar to the case without com- 
pensation of Fig. ^ the broadening is not symmetrical 
with respect to the mean residual nonlinear phase shift 
< < i ) RES> l|67|). With small mean nonlinear phase shift 
of <4>nl>= 1 rad, the received phase spreads further in 
the positive phase than the negative phase. With large 
mean nonlinear phase shift of < < I ) nl>= 3 rad, the re- 
ceived phase spreads further in the negative phase than 
the positive phase. The difference in the spreading for 
small and large mean nonlinear phase shift is due to the 
dependence between the residual nonlinear phase noise 
and the phase of amplifier noise. 



C. Error Probability of PSK Signals 



*cm = n - (* - aY) , (63) 

Ps+2 

The compensated received phase (|63f) is confined to 
the range of [— ir, +%). The p.d.f. of the compensated 
received phase is a periodic function with a period of 2ir. 
If the characteristic function of the compensated received 
phase is ^# om (^), the p.d.f. of the compensated received 
phase has a Fourier series expansion of 



l 

2^ 



or 



E 

m— — oo 



i l + °° 

P*c(0) = 7T + - E ^ {**™M exp(jm0)} 

Z7T 7T » — ' 



$cj OT (to) exp(jTO^). 



(64) 



(65) 



where 3?{-} denotes the real part of a complex number. 

Using the joint characteristic function of 
V?$,y,e n { v , m) Ip31)l. similar to that of (|r]2*)l . from 
the compensated received phase of ((6311 . the Fourier 
series coefficients are 



Assume that the decision angles are ±^7r — 9 C with the 
center phase of 9 C , the error probability is 



Pe,PSK = 1- / P* cm (0)d0, (68) 

or, similar to the error probability of ll4T|) . 

(69) 

From both (|66|l and l|40|) . the coefficients for the error 
probability (|oT?|l are 



n (2fe + l) = ^P^ e -A, + %^ 



2A* 



h 



A. 



Ik+l 



Afc,, 



(2fc+ 1) <$NL>" 
Ps + 1 

k > 0, 

(70) 



where, from 132(1 and 1(37(1 . 



^*cm ( m ) = ^<&,Y,d 



m <<1>NL> TO <$NL> 

, a : , 777, 



Ps + 2 Ps 4 

(66) 

Using the scale factor a m j n Fig. ED shows the 

p.d.f. of the compensated received phase (|rJ5|) with mean 
nonlinear phase shift of < $nl >= 0,1,2, and 3 rad. 
Shifted by the mean residual nonlinear phase shift of 



<$NL> , 

<3>res>= — T 



>, 



(67) 
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j(2fc+l)<*NL>~ 
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sin |2 


"j(2fe+l)<* NL >" 









(71) 



Aft 



1 + a 



j(2fc+l)<$ NL > 



tan 



j(2fc+l)<$ NL > 
T±I 



(72) 
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FIG. 14: With linear compensation, the error probability of 
PSK signal p e ,psK as a function of SNR p a . 



and ^$(^) (|16f> is the marginal characteristic function of 
nonlinear phase noise that depends solely on SNR. 

Based on the MMSE criterion, the center phase in l|69|) 
is C —<^kes> <|67[l and the scale factor in l|72l) is a = 
ttmin (JSnj) • Because the series summation of ljfi*9*t is a 
simple expression, numerical optimization can be used to 
find the linear MAP compensator to minimize the error 
probability. 

If the residual nonlinear phase noise is assumed to be 
independent to the phase of amplifier noise, similar to 
(|50|l . the error probability based on MMSE criterion can 
be approximated as 



1 _££. IP. 

Pe,PSK ~ - e ' 



(-1)* 
7T ^ 2k + 1 

(2k + 1) <$nl> 



-j(2fc + l)<5> RE s> 



m 



(73) 



where the characteristic function (v) is the char- 

acteristic function of (|62[1 with a — a m - ln (|59|) . 

Fig. E| shows the exact (|69[) and approximated i|73|) 
error probabilities as a function of SNR p s when nonlin- 
ear phase noise is compensated using the linear MMSE 
compensator. The error probability l|69|l is also mini- 
mized based on the MAP criterion and shown in Fig. El 
Fig- El also plots the error probability without nonlinear 
phase noise. 

Fig. El shows the SNR penalty of PSK signal for an 
error probability of 10 -9 calculated by the exact i|69|) 
and approximated (|73[1 error probability formulae with 
the linear MMSE compensator. The SNR penalty with 
the linear MAP compensator is also shown in Fig. IT51 




Mean Nonlinear Phase Shift <* NL > (rad) 



FIG. 16: The optimal center phase corresponding to the op- 
erating point of Fig. I15l as a function of mean nonlinear phase 
shift <$ NL >. 



The corresponding optimal center phase and scale factor 
of Fig. El are shown in Fig. El 

Fig-Elalso shows the SNR penalty of PSK signal with- 
out compensation from Fig. fusing the exact error prob- 
ability with optimal center phase there. For the same 
SNR penalty, the mean nonlinear phase shift with com- 
pensation is slightly larger than twice of that without 
compensation from Fig. 

The discrepancy between the exact and approximated 
error probability is smaller for small and large mean non- 
linear phase shift <$nl>- With the MMSE criterion, the 
largest discrepancy between the exact and approximated 
SNR penalty is about 0.37 dB at a mean nonlinear phase 
shift <<1>nl> around 2.53 rad. Like the conclusion of 
Sec. E w ith° u t compensation, with the MMSE criterion, 
the approximated error probability l|73(l is not accurate 
enough for linearly compensated PSK signals. 

Fig. 1151 also shows that the linear MMSE compensator 
using the optimal scale factor of l|59|l does not perform 
well as compared with the linear MAP compensator. The 
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largest discrepancy is about 0.34 dB at mean nonlinear 
phase shift of < $nl >= 1-68 rad. The major reason 
of this large discrepancy is due to the non-symmetrical 
p.d.f. of Fig.El 

Using the linear MAP compensator, the mean nonlin- 
ear phase shift <<I ) nl> must be less than 2.30 rad for 
a SNR penalty less than 1 dB, slightly more than twice 
that of Fig. of 1 rad without compensation. The opti- 
mal operating level is for a mean nonlinear phase noise 
< < J ) nl> of about 2.15 rad, slightly less than twice that 
of Fig. of 1.25 rad without compensation. 

From the optimal center phase from Fig. El for the 
linear MAP compensator, the optimal center phase is 
less than the mean nonlinear phase shift <<I>r,es> when 
the mean nonlinear phase shift is less than about 2.68 
rad. The changing of the optimal center phase with the 
mean nonlinear phase shift is consistent with Figs. and 
[T31 Similar to Fig. [7| when the residual nonlinear phase 
noise is assumed to be independent of the phase of ampli- 
fier noise using {73J, the optimal center phase is always 
larger than the mean residual nonlinear phase shift. The 
approximated error probability (|73|l is not useful to find 
the optimal center phase. 

The optimal scale factor from Fig. El is also not equal 
to the optimal MMSE scale factor (|59(l except when the 
mean nonlinear phase shift is very large. From Figs. Hoi 
and ED the MMSE criterion is not close to the perfor- 
mance of the optimal linearly compensated PSK signals 
based on MAP criterion. 



D. Error Probability of DPSK Signals 

A compensated DPSK signal is demodulated using in- 
terferometer of Fig.[S]to find the compensated differential 
phase of 



A$ cm = $ cm (t)-$ cm (t-T) 

= e n (t) - $resM - e„(t - t) + $ RES (t - t),(74) 

where $ C m( - )> ©n(")> an d $res(0 are the compensated 
received phase, the phase of amplifier noise, and the 
residual nonlinear phase noise as a function of time, and 
T is the symbol interval. The phases at t and t — T are 
independent of each other but are identically distributed 
random variables similar to that of l|41|) and (|63|l . The 
differential phase of (|74|l assumes that the transmitted 
phases at t and t — T are the same. 

From the p.d.f. of (|65[) . similar to H53|) . the p.d.f. of 
the differential compensated phase (|74|l is 



PA$ cm (6») 



2tt 



-E 



**c»0™)l cos(m0), (75) 



that is symmetrical with respect to the zero phase. 

Fig. El shows the p.d.f. of the differential received 
phase (|75|l with mean nonlinear phase shift of <$nl>= 
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FIG. 17: The p.d.f. of the differential compensated received 
phase PA* cm (#) in logarithmic scale. The inset is the same 
p.d.f. in linear scale. 



0,1,2, and 3 rad using the linear MMSE compensator 
with a — a m in l|59|l . The p.d.f. is plotted in logarithmic 
scale to show the difference in the tail. The same p.d.f. is 
plotted in linear scale in the inset. Fig. El is plotted for 
the case that the SNR is equal to p s = 20 (13 dB). From 
Fig. El when the p.d.f. of differential phase is broadened 
by the residual nonlinear phase noise, the broadening is 
symmetrical with respect to the zero phase. 

Similar to l|54|) and (|69|l . the error probability for 
DPSK signal is 



Pe,DPSK 



1 _ 2+^ 



(-If 



7T ^ 2k 

k=0 



* <I)cm (2fc + l)| z 



(76) 



where the coefficients of fy<s> crn (2k + 1) are given by l|70|) 
with parameters from ij7T|) and (j72"|l . 

Similar to the approximation for PSK signal l|73l) . if 
the nonlinear phase noise is assumed to be independent 
to the phase of amplifier noise, the error probability of 
(17611 can be approximated as 



Pe,DPSK 



p s e 



-Pb 



E 

k=0 



(~l) fc 

2k + 1 
(2k 



1) <*NL> 



.(77) 



Fig. El shows the exact (|76fl and approximated l|77|l 
error probabilities as a function of SNR p s for DPSK 
signal with linear MMSE compensator. The scale factor 
in (I72|) is also numerically optimized to find the linear 
MAP compensator to minimize the exact error proba- 
bility (ESJl. From Fig. El the linear MMSE and MAP 
compensators do not have big difference for linearly com- 
pensated DPSK signal. Fig. El also plots the error prob- 
ability without nonlinear phase noise. The approximated 
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SNR p_. (dB) 

FIG. 18: With linear compensation, the error probability of 
DPSK signal as a function of SNR p 3 . 




Mean Nonlinear Phase Shift <* > (rad) 



FIG. 19: With linear compensation, the SNR penalty of 
DPSK signal as a function of mean nonlinear phase shift 
<$nl>. 

error probability in Fig. is the same as that in but 
calculated for large number of fiber spans. From Fig. 1181 
the approximated error probability (j77(l always slightly 
overestimates the error probability. 

Fig. HI shows the SNR penalty of DPSK signal for an 
error probability of 10 -9 calculated by the exact (|76[) and 
approximated (|77(l error probability formulae for DPSK 
signal with linear MMSE and MAP compensation. The 
SNR penalty given by the approximated error probabil- 
ity (|77|l is the same as that in 9] but for large number of 
fiber spans. The discrepancy between the exact and ap- 
proximated error probability with either MMSE or MAP 
criteria is very small. The largest discrepancy between 
the exact and approximated SNR penalty with MMSE 
criterion is about 0.1 dB at a mean nonlinear phase shift 
< < I > nl> of 1.74 rad. Both with the exact error probabil- 



ity (|76|l . the linear MMSE and MAP compensators have 
the largest discrepancy of 0.06 dB at a mean nonlinear 
phase shift <$nl> of 0.95 rad. 

Fig. HI also shows the SNR penalty of DPSK signal 
without compensation from (fill) usingthe exact error 
probability formula there. Similar to [9J, for the same 
SNR penalty, DPSK signal with linear compensator can 
tolerate slightly larger than twice the mean nonlinear 
phase noise without linear compensation. 

For a power penalty less than 1 dB, the mean nonlinear 
phase shift <$nl> must be less than 1.31 rad, slightly 
larger than twice that of Fig.^Jof 0.57 rad without com- 
pensation. The optimal operating level of the mean non- 
linear phase shift <$nl> is about 1.81 rad such that the 
increase of power penalty is always less than the increase 
of mean nonlinear phase shift, slightly smaller than twice 
that of Fig. ^2 °f 1 racl without compensation. 

VII. NONLINEAR COMPENSATION OF 
NONLINEAR PHASE NOISE 

This section finds the optimal MAP and MMSE com- 
pensators for a PSK signal with nonlinear phase noise. 
We will first derive the joint p.d.f. of received ampli- 
tude and phase of PR.$ T .\g ('f, 9) given 9q is the trans- 
mitted phase. The MAP detector is derived through 
the Neyman-Pearson criterion 0, Sec. 5.3]. The er- 
ror probability is then calculated based on the p.d.f. of 
P-R,4v|o( r ) Based on the characteristic function of non- 
linear phase nosie condition on the received intensity, the 
optimal MMSE detector for signals with nonlinear phase 
noise is also derived analytically. 

A. Joint Distribution of Received Amplitude and 
Phase 

The received phase of (|41|) is confined to the range 
of [— ir,+ir). The joint PDF of received amplitude and 
phase Pn^ r \g Q (r, 9) can be modeled as a periodic function 
of 9 with a period of 2n and expanded as a Fourier series 
as 

P*..,|*M) = PJ £r + - E »{Cm(r)e*»< 9 -*)} , 

m— 1 

r > 0, (78) 

where Pn(r) is the PDF of received amplitude of i|28|l . 
C m (r) is the mth Fourier coefficient as a function of the 
received amplitude r, and 3?{-} denotes the real part of a 
complex number. The PDF of l|78[) has been simplified 
using the relationship of C_ m (r) = C* n (r). It is also 
obvious that PR,<s> r \e {r, 9)d9 = p R (r). 

As the Fourier series of the PDF PR^ r \e (r, 9) l(75)l. the 
Fourier coefficients of C m (r) are the Fourier transform 
of the PDF H78fl with respect to 9 at integer "angular 
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frequency" of to. The Fourier transform of a PDF is 
its characteristic function and C m (r) can come from the 
"partial" Fourier transform of the joint PDF of nonlinear 
phase noise, received intensity, and phase of amplifier 
noise. The Fourier coefficients are [8(j 

C m (r) = 4,ii,e n (^ m J~~^Jj2 > r > m ) ' ( 79 ) 
where q$ t R t o n {y, r, m) cannot be found directly but 

q^,Y,e n {^,y,m) = T~ x {**,y,e„} (v,V,m) (80) 

given by (|35|1 with f^^^n ( v -> to) given by 14Ufl and 
60 = 0. With a change of random variable of Y = R 2 , the 
partial PDF and characteristic function of (|3B|> becomes 



crt \ 2af, ) 



for m > and q$ R Q n (i/, r, — m) = q$ i R t e n (v, r, to). 
Based on (|79fl . we obtain 



(81) 



C m (r) 
with 



■ exp 



2s, 



<$NL> 



p s + 1/2 y ' 



/p s sec 



jto 



<$NL> 

/0 S + 1/2 



TO > 1, 

(82) 

(83) 
(84) 




and 
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FIG. 20: The distribution of received electric field for binary 
PSK signal. The contour lines are in logarithmic scale and 
both x and y axes are normalized to unity mean amplitude. 
The dashed line is the decision boundary. The SNR is p a = 18 
and the mean nonlinear phase shift is <$nl>= 2 rad. 



B. Optimal MAP Detector 

For a PSK signal with 6*o £ {0, 7r}, using the Neyman- 
Peason lemma with unity likelihood ratio 44, Sec. 5.3], 
the optimal decision regions of the MAP detector are 
given by 

K = {r,0 | p R ^ r \ (r,e) > p R ^ r \ w (r,9)\, (86) 
fti = {r,0 I PR,* r \«(r,6) >p R ,<i, r \o(r,e)} (87) 

for 9q = and 9$ = 7r, respectively. The error probability 
is 



2 V p s 



<$NL> 



1/2 



tan 



jto 



<$NL> 

Ps + 1/2 



where ty$(i>) is the characteristic function of nonlinear 
phase noise of (|16fl . Using method in quantum field the- 
ory, a PDF similar to H78|) was derived in |8l|. 

For a PSK signal with 8 £ {0,7r}, Figure I2T31 shows 
the distribution of the received electric field similar to 
Fig.^b). The contour lines of Fig.[5D]are the logarithmic 
of the PDF of i [pR,<s> r \o{r,9) + p R ^ r ^(r,9)] after the 
conversion to rectangular coordinate. The SNR of p s = 
18 is chosen for an error probability of 10~ 9 for binary 
PSK signal without nonlinear phase noise. The mean 
nonlinear phase shift of < <£>nl >= 2 rad is chosen to 
about double the optimal operating point estimated in 

m 

Similar to those of Fig. ^ the helix shape distribution 
of Fig.[20]clearly shows that the rotation due to nonlinear 
phase noise is correlated with the amplitude (the distance 
to the origin). The optimal MAP detector is derived here. 



Pe = \ [ P.R,* f .|,r(r,0)drd0 + - / p R ^ r \ (r, 8)drd6 
PR,<s> r \7r{r,0)drd9 



IiZi 



PR,$ r \o(r, 0)drd6. 



The decision regions of TZq and 1Z\ have a boundary of 
c (r) ± 7r/2, r > 0, where 9 c (r) is the center phase of the 
decision regions of IZo. With the joint PDF of (J7SJ, the 
error probability for MAP detector of H88fl becomes 



*-5-;E^jf«{«wM^'** w }*- 

fc=0 Jl> 

(89) 

Using only p Rt & r \o(r,0), because PR^ r \^{r,9) = 
PR,$ r \o( r i@ — 7r), the center phase of 9 c (r) can be de- 
termined by 
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o(r,0 c (r) 



= PR,<£ 



P |o(r,d c (r)-|). (90) 



Figure [5D| also shows the decision boundary of c (r) ± 
7r/2 based on i|86|) and (|87|l with center phase given by 
(|90|l . The decision boundary is the "valley" between two 
peaks of the PDF of PR^ r \o{r, 9) and PR^^ir, 9), re- 
spectively. 

As discussed earlier, the optimal MAP detector can be 
implemented based on the decision boundary of Fig. 1201 
resemble to the Yin- Yang logo of Chinese mysticism, 
called the Yin- Yang detector in j3f|. The same MAP 
detector can also be implemented as a nonlinear com- 
pensator by adding an angle of 9 c (r) from l|90|l to the 
received phase. The Yin- Yang detector is equivalent to 
the nonlinear compensator. 

In this paper, the optimal MAP detector is derived 
based on 9 c {r) as a function of the received amplitude. 
Because the relationship between intensity and ampli- 
tude is a monotonic function of Y — R 2 , R > 0, the non- 
linear phase noise is considered to be compensated by the 
received intensity instead of the received amplitude. 

The more popular DPSK signals can also be compen- 
sated using 9 c (ri) — 9 c (r2), where r\ and r-i are the re- 
ceived amplitudes in two consecutive symbols. While the 
compensator for DPSK signals can be constructed us- 
ing the center phase of H90|) . the evaluation of the error 
probability of nonlinearly compensated DPSK signals is 
difficult. 



C. Optimal MMSE Detector 

The optimal MMSE compensator estimates the nor- 
malized nonlinear phase using the received intensity by 
the conditional mean of E{<f>\R} = E{<f>\Y} = E{<I>\R 2 } 
|43|. The conditional mean is the Bayes estimator that 
minimizes the variance of the residual nonlinear phase 
noise without the constraint of linearity |44L Sec. 10.2]. 
We call the estimation of E{$\R} an estimation based 
on received intensity although it is actually based on re- 
ceived amplitude. To find the conditional mean E{&\R}, 
either the conditional PDF of p$>\r{9) or the conditional 
characteristic function of ^^r{v) is required. The con- 
ditional characteristic function of ^$>\r(v) can be found 
using the relationship of 



(91) 



py(y) 

where q$,Y(v, y) iPSt is the partial characteristic function 
of normalized nonlinear phase noise and PDF of received 
intensity and py (y) H27fl is the PDF of received intensity. 
Using J2HJ) and (J57J, we obtain 



^f^(v) exp 



^0 vlf 



2al exp [-{y + p s )} I (2Jyp~ s 



(92) 



The optimal MMSE compensator is the conditional 
mean of the normalized nonlinear phase noise given the 
received intensity of Y = R 2 , we obtain 



E{$\R} 



■ d T 
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+ r 4 



z/— 0,y— r 2 

Jplr Ii(2r^/p~ s ) 
3 7 (2rVpJ)' 



(93) 



Other than the normalization, the optimal MMSE com- 
pensator of H93fl is similar to that of |43 for finite number 
of fiber spans. The optimal MMSE compensator l|93l) de- 
pends solely on the system SNR. 

The normalized residual nonlinear phase noise is <p e = 
(\> — E{$\R}, its conditional variance is 



E{<f>\R} 2 



l + 4(p s + r 2 ) 



r 2 p s 



+ - 



90 9 
fp~ s r h{2rjpl) r 2 p s I 2 (2r^p~;) 



45 / (2r\/p7) 9 U(2r^p;) 



(94) 



The variance of the normalized residual nonlinear 
phase noise can be numerically integrated as 



„2 



'*e| 



R {r)PR(r)dr, 



(95) 



where pr(t) is the PDF of the received amplitude of 12811 . 

Similar to the error probability of (|89|l . the error prob- 
ability for MMSE detector is 



Pe = 



2 7T 

x exp 



(-1) 
2fc + l Jc 



fc=0 

j{2k + l) <$ NL > 



1/2 



>dr. 



(96) 



The factor of <$nl>/(Ps + 1/2) scales the normalized 
nonlinear phase noise $ in i|93|) to the actual nonlinear 
phase noise $nl- 

Figure |^ plots the standard deviation (STD) of the 
normalized nonlinear phase noise with and without com- 
pensation as a function of SNR p s . The STD of residual 
nonlinear phase noise with linear and nonlinear compen- 
sator are shown as almost overlapped solid and dotted 
lines, respectively. The STD of nonlinear compensator 
is about 0.2% less than that of linear compensator. Fig- 
ure |21 confirms the results of that linear and non- 
linear MMSE compensator performs the same in term of 
the variance of residual nonlinear phase noise. 
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SNR, p s (dB) 



FIG. 21: The STD of normalized nonlinear phase noise with 
and without compensation. The STD with linear and non- 
linear compensator are showed as dotted- and solid lines, re- 
spectively. 
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Received Intensity r/p s 

FIG. 22: The compensated phase as a function of received 
intensity. System parameters are the same as that in Fig. H3 



D. Numerical Results 

Figure l2~2l plots the center phase of 9 c {r) from l|90() as 
a function of the received amplitude of r 2 . The system 
parameters of Fig. [22J are the same as that of Fig. [201 
As discussed earlier, the center phase of Yin- Yang de- 
tector is the same as the compensated phase of a com- 
pensator. The center (or compensated) phase of the 
nonlinear MMSE compensator of l|93[) is also plotted in 
Fig. [221 for comparison. The phase of (|93[l is scaled by 
<$nl> l{p s + h)- The compensated phases of the linear 
compensator designed by MMSE or MAP criteria of Sec. 
IVII are also plotted in Fig.[221as dashed-lines for compar- 
ison. In Fig. 1221 the received intensity is normalized with 
respect to the SNR of p s . 

From Fig. [221 nonlinear MMSE or MAP compensated 
curves are very close to the linear compensated curves, 
especially when the received intensity is near its mean 




SNR p s (dB) 

FIG. 23: Error probability of a PSK signal with optimal MAP 
or MMSE detector. 

value of about r 2 /p s = 1. When the STD of Fig. [22 
is evaluated, the STD is mainly contributed from the 
region where the random variable is close to its mean 
value. Compared the linear and nonlinear compensated 
phases of Fig. [221 the STD of Fig. [^ for linear and non- 
linear MMSE compensator should not have significant 
difference. From Fig. [2J the linear and nonlinear MAP 
compensated phases are also very close to each other. 
In the region closes to r 2 /p s — 1, because the nonlin- 
ear MAP compensated phase is closer to the linear MAP 
compensated phase than the nonlinear MMSE compen- 
sated phase, the linear MAP compensated phase has bet- 
ter performance than the nonlinear MMSE compensated 
phase. 

Figure[2Slshows the error probability given by (|89|) and 
for optimal MAP and MMSE detectors, respectively, 
as a function of SNR p s . Figure 12*31 also plots the error 
probability of a PSK signal without nonlinear phase noise 
of p e = ierfc^/pj. From Fig. EU the optimal MMSE 
detector does not minimize the error probability. The 
optimal compensated phase of (|9"U|l always gives a smaller 
error probability that the compensated phase of l|93l) . 

Fig. [2H shows the SNR penalty of PSK signal for an 
error probability of 10~ 9 calculated by the MAP and 
MMSE (EH) error probability formulas. The SNR penalty 
with linear MMSE and MAP compensator of Fig. ^1 is 
also plotted as dashed lines for comparison. The SNR 
penalty without compensation Fig. is also shown for 
comparison. 

From Fig. [21 the nonlinear MMSE compensator per- 
forms up to 0.23 dB better than the linear MMSE com- 
pensator. Although Fig. [2J shows that linear and non- 
linear MMSE compensators performs almost the same in 
term of the STD of residual nonlinear phase noise, the 
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FIG. 24: The SNR penalty of PSK signal as a function of 
mean nonlinear phase shift <"! > nl>. 

error probability has significant difference. The compar- 
ison between Figs. [21 and [Ml shows that the variance 
does not correlate well with the error probability. From 
Fig. [21 the optimal linear MAP detector performs very 
close to the optimal nonlinear MAP detector. The op- 
timal nonlinear MAP detector performs only up to 0.14 
dB better than the linear MAP detector. 

Table [I] shows the mean nonlinear phase shift corre- 
sponding to a system with 1-dB SNR penalty and the 
optimal operating point. The optimal operating point is 
found by the condition that the increase of SNR penalty 
is less than the increase of SNR that is proportional to 
the mean nonlinear phase shift. Because of the steepness 
of the slope, the optimal MAP detector actually gives 
smaller optimal operating point than other compensa- 
tion schemes. 



VIII. CONCLUSION 

The joint characteristic functions of the nonlinear 
phase noise with electric field, received intensity, and the 



phase of amplifier noise are derived analytically the first 
time. The nonlinear phase noise is modeled asymptot- 
ically as a distributed process for large number of fiber 
spans. Replacing the span by span summation of the 
nonlinear phase noise by an integration, the distributed 
assumption is valid if the number of fiber spans is larger 
than 32. Using the joint characteristic function of the 
nonlinear phase noise with the phase of amplifier noise, 
the error probabilities of PSK and DPSK signal are cal- 
culated as a series summation. 

For PSK signals, the optimal decision region is not cen- 
tered with respect to the mean nonlinear phase shift. The 
dependence between linear and nonlinear phase noise in- 
creases the error probability of the signals. When the 
received intensity is used to compensate the nonlinear 
phase noise, based on the principle to minimize the vari- 
ance of the residual nonlinear phase noise, the optimal 
linear and nonlinear compensators are derived analyti- 
cally using the joint characteristic function of the non- 
linear phase noise with the received intensity. Using the 
exact error probability of systems with linearly compen- 
sated nonlinear phase, linear MAP compensator is calcu- 
lated based on numerical optimization. Using the distri- 
bution of a received signal with nonlinear phase noise, the 
optimal MAP detector is derived for a phase-modulated 
signal to minimize the error probability. The error proba- 
bility of nonlinear MAP and MMSE detector is also eval- 
uated using the distribution of the received signal. 

Having the same variance of residual nonlinear phase 
noise, nonlinear MMSE compensator performs up to 0.23 
dB better than linear MMSE compensator. The optimal 
nonlinear MAP compensator performs very close to the 
optimal linear MAP compensator with a difference less 
than 0.14 dB. While the MMSE criterion does not pro- 
vide a minimum error probability, the linear MAP com- 
pensator optimized by numerical methods is able to well 
approximate the optimal nonlinear detector. In practice, 
the optimal detector can be implemented as a Yin- Yang 
detector or a nonlinear compensator. 
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